ARTMAP: Supervised real-time learning and classification of nonstationary data by a self-organizing neural network

-This article introduces a new neural network architecture, called A R T M A P , that autonomously learns to class(~v arbitrarily many, arbitrarily ordered vectors into recognition categories based on predictive success. This supervised learning system is" built up from a pair of Adaptive Resonance Theory modules (ART, and ARTh) that are capable o f self-organizing stable recognition categories in response to arbitra O' sequences o f input patterns. During training trials, the ART, module receives a stream [a ~pj] of input patterns, and ARTh receives a stream [b ~p~] o f input patterns, where b ~ is the correct prediction given a ~p~. These A R T modules are linked by an associative learning network and an internal controller that ensures autonomous system operation in real time. During test trials', the remaining patterns a ~'~ are presented without b ~p~, and their predictions at ARTb are compared with b(pL Tested on a benchmark machine learning database in both on-line and o f f line simulations, the A R T M A P system learns orders" o f magnitude more quickly, efficiently, and accurate(v than alternative algorithms, and achieves 100% accuracy after training on less than half the input patterns in the database. It achieves these properties by using an internal controller that conjointly maximizes predictive generalization and minimizes predictive error by linking predictive success to category size on a trial-by-trial basis', using only local operations. This computation increases the vigilance parameter p~, of ART~ bv the minimal amount needed to correct a predictive error at A R TI,. Parameter p,, calibrates the minimum confidence that A R T, must have in a category, or hypothesis, activated by an input a ~r~ in order for ART~, to accept that category, rather than search for a better one through an automatically controlled process o f hypothesis testing. Parameter p,, is compared with the degree o f match between a ~p> and the top-down learned expectation, or prototype, that is read-out subsequent to activation o f an ART~, category. Search occurs i f the degree o f match is less than p~,. A R T M A P is hereby a (vpe o f self-organizing expert system that calibrates the selectivity o f its hypotheses based upon predictive success. As a result, rare but important events can be quickly and sharply distinguis'hed even if the)' are similar to frequent events with different consequences. Between input trials p~, relaxes to a baseline vigilance 7,,. When P., is large, the system runs in a conservative mode, wherein predictions are made only it" the system is" confident o f the outcome. Very few Jalse-alarm errors then occur at any stage o f learning, yet the system reaches asymptote with no loss of speed. Because A R TMA P learning is self-stabilizing, it can continue learning one or more databases, without degrading its corpus o f memories, until its Jull memory capacity is utilized. Keywords--ARTMAP, Adaptive resonance theory, Supervised learning, Self-organization, Prediction, Expert system, Mushroom database, Machine learning. ? Supported in part by BP (98-A-1204), DARPA (AFOSR 90(t083), and the National Science Foundation (NSF IRI-90-00539). $ Supported in part by the Air Force Office of Scientific Research (AFOSR 90-(1175 and AFOSR 90-0128), and DARPA (AFOSR 90-0083). § Supported in part by DARPA (AFOSR 90-0083). Acknowledgements: The authors wish to thank Cynthia E. Bradford for her valuable assistance in the preparation of the manuscript. Requests for reprints should be sent to Professor Gail Carpenter, Center for Adaptive Systems, Boston University, 111 Cummington Street, Boston, MA 02215. 565 1. I N T R O D U C T I O N : P R E D I C T I V E A R T As we move freely through the world, we can at tend to both familiar and novel objects , and can rapidly learn to recognize, test hypotheses about , and learn to name novel objects without unselectively disrupting our m e m o r i e s of familiar o b j e c t s This article describes a new self-organizing neural ne twork arc h i t e c t u r e c a l l e d a Predict ive A R T or A R T M A P arch i tec ture that is capable of fast, yet stable, online recognit ion learning, hypothesis testing, and adapt ive naming in response to an arbitrary s tream of input patterns. 566 G. A, Carpenter, S. Grossberg, and ,1. H. Rewudds The possibility of stable learning in response to an arbitrary stream of inputs is required by an autonomous learning agent that needs to cope with unexpected events in an uncontrolled environment. One cannot restrict the agent's ability to process input sequences if one cannot predict the environment in which the agent must successfully function. The ability of humans to vividly remember exciting adventure movies is a familiar example of fast learning in an unfamiliar environment. 1.1. Fast Learning About Rare Events A successful autonomous agent must be able to learn about rare events that have important consequences, even if these rare events are similar to frequent events with very different consequences. Survival may hereby depend on fast learning in a nonstationary environment. Many learning schemes are. in contrast, slow learning models that average over individual event occurrences and are degraded by learning instabilities in a nonstationary environment (Carpenter & Grossberg, 1988; Grossberg, 1988a). 1.2. Many-to-One and One-to-Many Learning An efficient recognition system needs to be capable of many-to-one learning. For example, each of the different exemplars of the font for a prescribed letter may generate a single compressed representation that serves as a visual recognition category. This exemplar-to-category transformation is a case of manyto-one learning. In addition, many different fonts, including lower case and upper case printed fonts and scripts of various kinds, can all lead to the same verbal name for the letter. This is a second sense in which learning may be many-to-one. Learning may also be one-to-many, so that a single object can generate many different predictions or names. For example, upon looking at a banana, one may classify it as an oblong object, a fruit, a banana, a yellow banana, and so on. A flexible knowledge system may thus need to represent in its memory many predictions for each object, and to make the best prediction for each different context in which the object is embedded. 1.3. Control of Hypothesis Testing, Attention, and Learning by Predictive Success Why does not an autonomous recognition system get trapped into learning only that interpretation of an object which is most salient given the system's initial biases? One factor is the ability of that system to reorganize its recognition, hypothesis testing, and naming operations based upon its predictive success or failure. For example, a person may learn a visual recognition category based upon seeing bananas of various colors and associate that category w~th ~, certain taste. Due to the variability of color features compared with those of visual form, this learned recognition category may incorp~rate form features more strongly than color features. However. the color green may suddenly, arid unexpectedly, become an important differential predictor of a banana's taste. The different taste of a green bandana triggers hypothesis testing that shifts the focus of visual a~tention to give greater weight, ~ ~aliencc. w, the banana's color features withou~ negating the importance of the other features ~hat define a bananas form A new visual recogniuon category can hereby form for green bananas. :lnd this category can be used to accurately predict ~he different taste of green bananas. The new, finer category can form. moreover, without recoding eithc~ the previously learned generic representation of !~ananas or their taste association. Future representations may also k)rm that incorporate new knowledge about bananas, without disrupting the representations that arc used to predict their different tastes. In this way, predictive feedback provides one means whereby one-to-many recogmtion and prediction codes can form through time. by using hypothesis testing and attention shifts that support new recognition learning without forcing unselective forgetting of prewous knowledge 1.4. Adaptive Resonance Theory The architecture described herein forms part of Adaptive Resonance Theory, or ART, which was introduced in 1976 (Grossberg, 1976a. 1976b) in order to analyze how brain networks can autonomously learn in real time about a changing world in a rapid but stable fashion. Since that time, ART has steadily developed as a physical theory to explain and predict ever larger data bases about cognitive information processing and its neural substrates IGrossberg, 1982a. 1987a, 1987b. 1988b). A parallel development has described a series of rigorously characterized neural architectures called ARq 1. ART 2. and ART 3--with increasingly powerful learning, pattern recognition, and hypothesis testing capabilities (Car penter & Grossberg, 1987a. 1987b, 1988. 1990). 1.5. Serf-Organizing Predictive Maps The present class of architectures are called Predicuve ART architectures because they incorporate ART modules into systems that can learn to predict a prescribed m-dimensional output vector b given a prescribed n-dimensional input vector a (Figure 1). The present example of Predictive ART is called ARTMAP because its transformation from vectors